Skip to content

[2186] Scaling utils#2231

Open
florianscheidl wants to merge 72 commits into
ecmwf:developfrom
florianscheidl:ekfs/scaling-plots-20260417
Open

[2186] Scaling utils#2231
florianscheidl wants to merge 72 commits into
ecmwf:developfrom
florianscheidl:ekfs/scaling-plots-20260417

Conversation

@florianscheidl
Copy link
Copy Markdown
Contributor

@florianscheidl florianscheidl commented Apr 17, 2026 •
edited
Loading

Description

Contains utilities and configs we used for scaling experiments:

  • era5_georing config and required streams
  • elapsed time metrics in the trainer and run_train
  • scripts for extracting data from scaling experiments and generating plots/tables (moved into an optional package "weathergen-performance" (see README for how to use).

Example plots and tables:

jupiter_weak_scaling_table jupiter_weak_scaling

Small code modifications became necessary as we trained on more than 16 nodes. Specifically:

  • Lower bound on warmup, cooldown, and decay steps in lr_scheduler
  • Lower bound on beta2 used in Adam optimizer (negative values are illegal). Set to 0.9 now, based on Internet suggestion, but open to suggestions.
  • Lower bound on len_per_rank when setting mini_epoch_base (avoiding zero-division error for small configs).

Issue Number

Fixes #2186

Checklist before asking for review

  • I have performed a self-review of my code
  • My changes comply with basic sanity checks:
    • I have fixed formatting issues with ./scripts/actions.sh lint
    • I have run unit tests with ./scripts/actions.sh unit-test
    • I have documented my code and I have updated the docstrings.
    • I have added unit tests, if relevant
  • I have tried my changes with data and code:
    • I have run the integration tests with ./scripts/actions.sh integration-test
    • (bigger changes) I have run a full training and I have written in the comment the run_id(s): launch-slurm.py --time 60
    • (bigger changes and experiments) I have shared a hegdedoc in the github issue with all the configurations and runs for this experiments
  • I have informed and aligned with people impacted by my change:
    • for config changes: the MatterMost channels and/or a design doc
    • for changes of dependencies: the MatterMost software development channel

@florianscheidl
feat: add timing metrics (startup, training, overall)
644273a 
- Track startup_time_seconds: time from run() start to training loop
- Track total_training_time_seconds: time in training/validation cycles
- Track overall_time_seconds: total wall-clock time from launch to finish
- All metrics logged only on root rank to avoid file contention
- Metrics written to metrics.json, automatically uploaded to MLflow
- Console logs show timing summaries for quick monitoring
@florianscheidl
docs: add agent structure with skills, tasks, and docs
92120b8 
- Created .hermes/ directory with skills/, tasks/, docs/ subfolders
- Added skills overview (README.md) with task-type skills
- Implemented 'planning' and 'metrics' skills
- Documented timing metrics task in tasks/2026-04-17-timing-metrics/
- Added agent structure documentation
- Updated .gitignore with optional .hermes/ entry
@florianscheidl
docs: add skills review cycle for periodic compactification
df69dcb 
- Added 2-3 month review cycle recommendation
- Defined criteria for skill consolidation
- Included usage frequency thresholds
- Documented when to merge or remove skills
@florianscheidl
configs
7f0648f
florianscheidl and others added 25 commits April 17, 2026 15:58
@florianscheidl
Merge branch 'feature/timing-metrics' into ekfs/scaling-plots-20260417
8fe45a0
@florianscheidl
Remove hermes tool tracking for now
dd55fb0
@florianscheidl
Try duration metrics
09b6e82
@florianscheidl
Update metrics, store after each mini-epoch
da3c29b
@florianscheidl
Refactor configs/streams
fc9a111
@florianscheidl
Extract scaling data
cfc4c62
@florianscheidl
Script to generate scaling plots
82b503a
@florianscheidl
Script update
70053b1
@florianscheidl
Repeat data in mini epoch
0c2df97
@florianscheidl
corrected time window length
2c79d28
@florianscheidl
Merge branch 'ekfs/scaling-plots-20260417' of github.com:florianschei…
6374986 
…dl/WeatherGenerator into ekfs/scaling-plots-20260417
@florianscheidl
Lower to 512 samples per mini epoch
b5d70f6
@florianscheidl
Updated extraction script
f46828c
@florianscheidl
Merge branch 'ekfs/scaling-plots-20260417' of github.com:florianschei…
89ac519 
…dl/WeatherGenerator into ekfs/scaling-plots-20260417
@florianscheidl
Log time more often
7cad6b5
@florianscheidl
Fix training start scope
30ac102
@florianscheidl
Minimal validation
5e7f63e
@florianscheidl
Increase samples_per_mini_epoch to 1024
2be95c6
@florianscheidl
Final training duration and terminal/metric logging
93b203b
@florianscheidl
log metrics after mini-epoch
2b708e3
@florianscheidl
Log metrics after mini-epoch, change schema
0d8407d
@florianscheidl
MEtric typo
422fc60
@florianscheidl
Logging refactor
f63cba9
@florianscheidl
Update extraction script
b596c14
@florianscheidl
NNode extraction
42ba646
@florianscheidl florianscheidl changed the title [2186](WIP) scaling plots branch [2186](WIP) scaling utils May 4, 2026
@florianscheidl florianscheidl changed the title [2186](WIP) scaling utils [2186] Scaling utils May 4, 2026
@florianscheidl florianscheidl marked this pull request as ready for review May 4, 2026 09:49
@github-actions github-actions Bot added the performance Work related to performance improvements label May 4, 2026
ekouts
ekouts suggested changes May 8, 2026
View reviewed changes
Comment thread src/weathergen/train/trainer.py Outdated
@github-project-automation github-project-automation Bot moved this to In Progress in WeatherGen-dev May 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@ekouts ekouts ekouts approved these changes

Reviewers whose approvals may not affect merge requirements

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

performance Work related to performance improvements

Projects

WeatherGen-dev
Status: In Progress

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Create scaling plots for compute time proposal

2 participants

@florianscheidl @ekouts